From machine learning to data visualization, Big Data has developed a fairly mature technology tree, with different technical levels having different technical architectures, and new technical terms emerging every year.
In fact, to know what are the core technologies of big data is very simple, there are four aspects: big data acquisition, big data pre-processing, big data storage, big data analysis, which together form the most core technologies in the life cycle of big data: 1:
1、 Big data acquisition
Big data collection, that is, the collection of structured and unstructured massive data from various sources.
Database collection: popular Sqoop and ETL, traditional relational databases MySQL and Oracle also still serve as the data storage method for many enterprises.
Web data collection: A data collection method that obtains unstructured or semi-structured data from web pages with the help of web crawlers or public APIs of websites, and unifies and structures them into local data.
File collection: including real-time file collection and processing technology flume, ELK-based log collection and incremental collection, etc.
2、 Big data pre-processing
Big data pre-processing refers to a series of operations performed on the collected raw data before data analysis, aiming to improve data quality and lay the foundation for later analysis. Data pre-processing mainly includes four parts:
Data cleaning: refers to the use of cleaning tools such as ETL to process data with missing data (missing attributes of interest), noisy data (data with errors, or data that deviate from the expected value), and inconsistent data.
Data integration: It is a storage method that combines data from different data sources and stores them in a unified database, focusing on three problems: pattern matching, data redundancy, and data value conflict detection and processing.
Data conversion: It refers to the process of processing the inconsistencies in the extracted data. The abnormal data is cleaned according to business rules to ensure the accuracy of subsequent analysis results
Data Statute: It refers to the operation of streamlining the data volume to the maximum extent to get a smaller data set on the basis of maintaining the original appearance of the data.
3、Big data storage
Big data storage, refers to the process of storing the collected data with memory, in the form of a database, contains three typical routes:
A new database cluster based on MPP architecture
Adopt Shared Nothing architecture, combined with the efficient distributed computing model of MPP architecture, through a number of big data processing technologies such as column storage, coarse-grained indexing, etc., focusing on the data storage methods unfolded for industry big data.
Hadoop-based technology extension and encapsulation
This is the process of using Hadoop open source advantages and related features to derive relevant big data technologies for data and scenarios that are difficult to be handled by traditional relational databases. The most typical application scenario at present: to realize the support of Internet big data storage and analysis by extending and encapsulating Hadoop, which involves dozens of NoSQL technologies.
Big Data All-in-One
This is a kind of combined software and hardware product designed for the analysis and processing of big data. It consists of a set of integrated servers, storage devices, operating systems, database management systems, and pre-installed and optimized software for data query, processing, and analysis, with good stability and vertical scalability.
4、Big data analysis and mining
This is the process of extracting, refining and analyzing the disorganized data, which can be divided into
Visualization analysis
Visualization analysis, refers to the graphical means to clearly and effectively communicate and communicate information analysis means. It is mainly applied to massive data correlation analysis, i.e. the process of correlation analysis of scattered heterogeneous data and making a complete analysis chart with the help of visual data analysis platform.
Data mining algorithm
Data mining algorithm, that is, by creating data mining model, and the data to try and calculate the data, data analysis means. It is the theoretical core of big data analysis. There are various data mining algorithms, and different algorithms present different data characteristics based on different data types and formats.
Predictive Analytics
Predictive analytics, one of the most important application areas of big data analytics, achieves the purpose of predicting uncertain events by combining various advanced analytic functions. It helps users analyze trends, patterns and relationships in structured and unstructured data, and use these metrics to predict future events and provide a basis for taking action.
Semantic Engine
Semantic engine, which refers to the operation of adding semantics to existing data to improve users' Internet search experience.
Data Quality Management
It refers to a series of management activities to identify, measure, monitor, and warn about various data quality issues that may arise in each phase of the data lifecycle, in order to improve data quality.